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(54) Classifying data in a database 

(57) The present invention relates to a method 
and apparatus for local Bayesian classification and 
regression. In a first step, a joint estimate is made of the 
density in the covariate and response variables consti- 
tuting multivariate normal distributions. Subsequently 
responses are sampled from the estimate and finally an 
arbitrary transfomiation is performed on latent parame- 
ters of a k-dimensional distribution, where each dimen- 
sion is defined on a real line (i.e - oo to + <») and where k 
is an integer greater than or equal to 2. 
The invention offers extremely flexible ways of model- 
ling data and is capable of dealing with a highly varied 
customer base that does not follow a global trend. It is 
unaffected by outliers, such as errors in the data or 
small groups of customers displaying unusual behav- 
iour. It can also deal elegantly with missing values in the 
data. An important technical contribution of the inven- 
tion is that it allows inferences to be reversed from a 
given behaviour to demographic descriptions of custom- 
ers that perfonn that behaviour. In other words It can 
provide a previously unobtainable executive summary 
describing the factors that drive customer behaviour. 
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Description 

[0001] The present invention relates to a metlioc! and apparatus for local Bayesian classification and regression. 
[0002] All predictive modelling is based on the idea that similar people will behave in a similar way. This means that 

5 a bank with a data warehouse that has information about who bought and did not buy a certain product can find neigh- 
bourhoods in the data space that are lil^ely prospects for a sale. Broadly speal<ing there are two ways of approaching 
the problem of prediction: global methods and local methods. Almost all models cun-ently In use for propensity model- 
ling are global models. These include neural networks, linear regression and logistic regression. These and other exam- 
ples are listed in Table 1. There are Bayesian versions of all of them, however, among Bayesian statisticians there has 

10 been a recent growth of interest in local models. This is because they seem to offer a more flexible and possibly more 
accurate way of approaching data modelling. 



15 


Global Models 


Local Models 




Linear regression 


Nearest neighbour 




Logistic regression 


Local polynomial regression 


20 




Multivariate Adaptive 


Back-propagation 


Regression Splines (MARS) 




nexu^ netwoiics 


Decision trees (CART and 




25 




C5,0) 



Table I. A list of local and global models. 



30 

[0003] In a global model, as the name suggests, an assumption is made about iiow the data is distributed as a 
whole and so global modelling is a search for global trends. Local models are not concerned at all with global trends. 
Instead they care about small neighbourhoods of similar customers. By looking after many local neighbourhoods of 
similar customers local models can build up global trends in a piecemeal fashion. The simplest example of a local model 
35 is the nearest neighbour algorithm. This model is so simple that it takes only a few lines of code to implement. It looks 
like this: 

Find most similar customer 
Behaviour is the same as that customer 

40 

[0004] The nearest neighbour algorithm only cares about the most similar customer's behaviour. There is no men- 
tion of all the other customers who are not similar to the customer for which we are making the prediction. This is the 
hallmark of local models. 

[0005] An example of a global model is logistic regression. In this model, a line is drawn through the space of cus- 
45 tomer information to classify customers in relation to whether they will or will not buy a particular product. Everyone who 
lies on one side of the line is not going to buy and everyone on the other side of the line will buy. This line is also called 
a partition, because it splits the input space in which the customers lie into segments. In a global back-propagation neu- 
ral network the customer data space is first warped to a new space and then a line is drawn in the new space to sepa- 
rate customers who buy from those that do not. That is, global models use a global trend to fit customer behaviour and 
50 assume that all customers will follow this trend. This can cause problems. 

[0006] The first problem is that all customers may not follow the global trend. Some sub-populations of customers 
may differ radically from others. So in an attempt to stay with a global model, it is necessary to make the global trend 
more complicated at greater computational cost. In a neural network, for example, it is possible to.add more neurons to 
the network to make the data partitions more convoluted. But this slows down the modelling process, and knowing just 
55 how many neurons to add is complex. 

[0007] Another problem with global models is outliers, if some of the data in the warehouse is incorrect, or just unu- 
sual, a global model will still blindly try to fit that data with a global trend. This means that the core customer data that 
really should fit the global trend will be compromised by having to accommodate the few outlying data points. A whole 
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branch of statistics is devoted to patching up global nnodels to get around the problenfi of outliers. 
[0008] If a lot of customers really are outliers, unless the global model is made very complex rt will predict poorly. 
One solution is to use a very powerful model such as a back-propagation neural net, with very many hidden units. How- 
ever, such models are very slow and fit clusters of outliers poorly if they have few members. It is also difficult to inten-o- 

5 gate a neural network to get a feel for how customers behave in terms a human can understand. 

[0009] It has already been suggested to use a Bayesian local model for classification and regression based on mix- 
tures of multivariate normal distributions. Each multivariate nonnal distribution in the model represents a cluster to 
which customers belong with a given probability. A result of using the multivariate normal distribution is that the target 
variable follows a linear trend, that is, we have linear regression within each cluster. 

10 [0010] Underlying the previous approach is a joint model of the input and output variables. The model uses a finite 
mixture of multivariate nornial distributions of dimension d + 1, where d is the number of input variables. This model 
leads to the conditional distribution (ylx) being a finite mixture of univariate nonnal distributions, say N(yrjly. Xj), with 
weights proportional to the likelihood of x coming from the corresponding marginal multivariate nomnal distribution, for 
example, N^(xlily, Kj). Formally this can be expressed as 

15 

{x,y)'-ZjN^^^(x,y\uXj) = ZjPj(x.y) 

= EyPy(X)N(yl^y.iy) 

20 

This is the same as local linear regression which can be expressed mathematically as E(/lx) = Ly Sj(x)mj{x) where 
my(x) denote distinct linear functions of x and sy{x) denote probability weights that vary across the sample space x. A 
limitation of the previously suggested model is that it can only handle continuous input and target variables. 
[001 1 ] An object of the present invention Is to deal with discrete and categorical input and output data. 
25 [001 2] According to the present invention, there is now provided a method of classifying discrete data in a database 
comprising the steps of; 

jointly estimating the density in the covariate and response variables constituting multivariate normal distributions, 
sampling responses from the estimate, and 
30 performing an artDltrary transformation on latent parameters of a k-dimensional distribution, where each dimension 
is defined on a real line (i.e - oo to + oo) and where k is an integer greater than or equal to 2. ■ 

[001 3J Further according to the present invention, there is provided data processing apparatus for classifying dis- 
crete data in a database the apparatus comprising; 

35 

means for jointly estimating the density in the covariate and response variables constituting multivariate normal dis- 
tributions, 

sampling means for sampling responses from the estimate, and 

transformation means for performing an arbitrary transformation on latent parameters of a k-dimensional distribu- 
te tion, where each dimension is defined on a real line (i.e - « to + ») and where k is an integer greater than or equal 
to 2. 

[0014] Yet further, according to the present invention, there is provided a computer program adapted for program- 
ming data processing apparatus to classify discrete data in a database the program comprising; 

45 

instructions for jointly estimating the density in the covariate and response variables constituting multivariate nor- 
mal distributions, 

instructions for sampling responses from the estimate, and 

instructions for perfomning an art^itrary transformation on latent parameters of a k-dimensional distribution, where 
50 each dimension is defined on a real line (i.e - «> to + oo) and where k is an integer greater than or equal to 2. 

[0015] The invention will be further described, by way of example, with reference to the accompanying drawings in 
which; 

55 Figure 1 shows a database relating to a set of customers of a bank, 

Figure 2 shows data processing apparatus, according to the present invention, for implementing the method of the 
invention, and 

Figure 3 shows a block diagram of the operating steps in a method, according to the present invention, which 
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employs the apparatus of Figure 3. 

[0016] In Figure 1, a database of information 10 concerns a set of customers of an enterprise such a bank. The 
database 10 is shown in three dimensions arranged along an X axis 1 1, a Y axis 12 and a Z axis 13. The X axis 1 1 is 

5 subdivided to Include fields for each of five customers A to E although It will be understood that in practice the database 
would have fields along the X axis for a multiplicity of customers far in excess of five. Each customer has attributes 
which include personal information such as age, type of job and salary and financial attributes such as the number and 
types of bank accounts, including investment accounts, deposit and current accounts and loans. The customer 
attributes are an-anged along the Z axis 1 3. Whilst only five fields of attributes for each customer are illustrated in Figure 

10 1, it will be understood that, in practice, provision would be made in the database to Include a number of attributes for 
each customer far in excess of five. 

[0017] The Y axis 12 of the database 10 is a time axis. Each customer attribute is recorded for each of a regular 
succession of time intervals. The time Intervals can conveniently be intervals of one month although intervals other than 
monthly intervals can be used. 

• 15 [0018] In Figure 2 is shown a data processing apparatus which provides a means for receiving the data in the data- 
base 10 of Figure 1 and for predicting the behaviour of the customers in the database thereby to enable the targeted 
marketing of products to customers judged most likely to buy the products. 

[0019] The data is entered by means of an I/O device 20 through an I/O port 21 for entry to a data bus 22. A group 
of central processing units (CPU's) 23 are controlled by an operating system 24 to control the entry of data onto the 

20 data bus. A random access memory 25 is connected to the I/O bus 22 to receive and store data supplied by the bus 22. 
[0020] . In response to the entry, by means of the data I/O device 20, of the data representing the information in the 
different fields of the database 1 0 of Figure 1 , the parallel operating CPUs are programmed to enter the data into the 
memory 25. The data in the memory 25 is a set of reference data sequences. The CPUs are programmed by computer 
programming, according to the present Invention, to process the data sequences in steps represented In the block flow 

25 diagram of Figure 3. 

[0021] Preparatory to discussing the block flow diagram of Figure 3, it will be helpful to discuss the mathematical 
basis on which the processing operations of the invention proceed. A mixture of multivariate normal distributions is con- 
structed to model the joint distribution of the input and target variables using the Dirichlet process mixtures of normals 
model. This is a Bayesian non-parametric model that enables a very flexible model of the joint distribution of (x, y) to be 

30 . fitted. Bayesian non-parametric models construct distributions on larger spaces than standard parametric models. They 
thus provide support for more eventualities such as multimodal or skewed distributions. In the following section we 
describe the basic Dirichlet mixtures of normals model. 
• [0022] The model assumes that the observed data vectors Zf are conditionally independent and sampled jointly 
from a multivariate notrnal distribution, Z/Itc/ Nrf{z/l/i/, A,), where d Is the dimension of z/ and 7i=(|i, X) . The parame- 

35 ters of the normal distribution (jt,, A/) are assumed to be sampled from an unknown distribution G , whteh we model as 
a Dirichlet process. The Dirichlet process is defined by two parameters, Gq, a parametric distribution which detennines 
the prior expectation of G and (|> which expresses our uncertainty about Gq. This model can be expressed mathemati- 
cally as a Bayesian hierarchical model as follows 

40 Zl-t^^{Zf\\ij.Xi)i = ^ n 

G~D{*Go) 

45 

Thus, in our model Gq defines the mean location of the 7t/s. The parameter ^ can be thought of as controlling the vari- 
ability of G about Go: smaller <p the greater the variability about Gq. 

[0023] For convenience we take Gq to be the conjugate prior distribution for the multivariate normal distribution with 
unknown scale and location. This is the normal Wishart distribution 

50 

Go = Nd(>*'J*o-"o^)Wic/(>'>a.P) 

A key feature of the model is the discreteness of the Dirichlet process. Let us consider the parameter values 
(fi;, Xi) = nf which we assume are drawn from an unknown distribution G to which we assign a Dirichlet process prior 
55 distribution. Given a hypothetical sample n of size n from G, there is a positive probability of coincident values, that is 
nf = njt i * j. This is evident from the conditional prior distribution. Let be k without ;r/, 
71 ^. = (71 , 71 , 71 7C „) then the conditional prior distribution of 7i/ is 
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(7t,l7C.;)--<>a„.iGo(7C;)+a„.i £ 5^^ (ti,) 

5 where 

10 

(n) denotes a unit point mass at tc = Jiy and = 1/((j>+n) . 
Similarly the distribution of a future parameter n„+^ is given by 

n 

/=1 



Thus, given a sample n of size n from G, will tal<e a new distinct value with probability (pap and will otherwise take 
20 a coincident value nj with probability a^. 

[0024] The values taken by ;ri will, with positive probability, reduce to k < n distinct values. Denote the k dis- 
tinct values of by 7C* = (^i*. X*) . Let nj denote the number of ;r/s taking the value n] . The full conditional prior distri- 
bution of nn+^ will thus reduce to the following mixture of fewer components 

25 k . 

30 The number of distinct values will increase stochastically with 0. The distinct values can be thought of as cluster cen- 
tres. 

[0025] Given the predictive distribution of the unobserved parameters n^^^^ we can evaluate the predictive distribu- 
tion of the observed data z„^i. 

35 (Z„^1 I n) ~ Jp(Zn+i t n„+i)P(Jln+1 ' '^)^'^n^^ 

The first term in the integrand is the multivariate normal distribution and the second tenn is given above, thus it follows 
40 (z^^, I %) ^ (fa„Sf^ [z„^T I Ho.^^ (a-l(d-1))(J-\2a-c/+ l) + a„ ][; n^N^{z^^, I n). x)) 



[0026] Thus, the posterior predictive distribution of z„+i having observed data D = (z^,..., z^), can be written as 

45 

p(z„^i I D) ~ Ip(z„^i I n)p{n\ D)chi 

This integral can be evaluated via a Monte Cario approximation given draws from the posterior distribution p(;rl Dp) as 
follows 

50 

/=1 

55 

where the ;r/s are drawn from the posterior distribution 
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[0027] In the following section we outline a connpLitational strategy to sample from p(n\D). 
Computational strategy 

5 [0028] The posterior distribution can be simulated via the Gibbs sampler algorithm. To implement the Gibbs sam- 
pler algorithm the full conditional distributions must be identified. The full conditional distribution for tc/ is given by 

k 

10 j=^,J*^ ' 



where G/(;r,) is the posterior distribution of nj given the observation X/ and the prior distribution Gq . Thus G/(jiy) is given 
by the following normal Wishart distribution 

15 

20 The weights qj are defined as follows 



30 where 



55 

y'=2Y-d+1 

[0029] Where the parameters a, y and /Iq are the parameters of the base prior, Gq, defined earlier. 
[0030] The weight qo is proportional to ^ times prior predictive density under the prior Gq evaluated at x/ and the 
40 weights qj are proportional to nj times the likelihood of N{A m , A.; ) evaluated at x/. The prior predictive distribution is 
defined as 

p(x)=J/V^(xln.X)Go(7i)dncfA. 
=Sftf{>flfio.P'.7') 



[0031] Here is the Gibbs sampler in algorithm form: 

50 Step 1 : Choose starting values for n=(7i ^ n „) ; reasonable starting values can be obtained by talcing samples 

from the individual conditional posteriors G/( * ). 

Step 2: Sample sequentially from (n-i I n.^, D), (712 I n.^, D) up to (n^ I n,^, D). 

55 Step 3: Return to 2 until convergence. 

[0032] The parameters of our prior distribution Gq can be learnt about through the data D. This involves assigning 
the parameters prior distributions and including draws from their full conditional distributions In the Gibbs sequence 



6 



3/23/2007, EAST Version: 2.1.0.14 



EP 1 085 429 A1 



described above. This model gives greater modelling flexibility with little added computational expense. 
Regression function estimation 

5 [0033] A continuous regression function E(y I x) is estimated. As outlined at the start of this section the input space, 
X, and the output space, y, are modelled jointly as a Dirichiet process mixture of multivariate normal distributions fol- 
lowing the methodology already described. Since y is univariate and if x Is a vector of length k, that is, we have k input 
variables, the multivariate normal distributions will be of dimension k +1 . Recall that the Gibbs sampler was used to sim- 
ulate from the posterior distribution of the unknown parameters n. At each sweep of this Gibbs sampler the values of 

10 the 7;*'s and the allocations of the n's to the 7c*'s is known. Given these values it is straightf onward to show that the con- 
ditional distribution of an unknown response, y^^i, given input variables, x^+i. is given by 

k 

15 y=i 



where fo is the conditional prior predictive distribution of y„^i given x„+i under the prior distribution Gq and is the con- 
ditional normal distribution of y^^^ given x„^.1 under the joint model 

20 

The weights qo and qrj are similar to those defined for the basic Dirichiet mixture of nonnais model except that the 
weights are conditioned on x^^^ and represent the probability that x^^.^ is from the prior predictive distribution under the 
25 prior distribution Gq and the joint model 

'Vrf+1{^^+1./^+1'^i^^■) 

respectively. 

30 [0034] The expectation of the regression function can be evaluated at each iteration of the Gibbs sampler as 

k 

E(ylx)=J^ qj{x)m^{x) 

35 

where mj(x) is the mean of the /-th component distribution for y given x. For both the Student and normal distribution 
mj(x) is a linear function of x. The closer x is to the y-th component the greater the weight qj, giving higher weight to the 
prediction from cluster J when x is In a region 'close' to cluster /. 
40 [0035] The average of these conditional expectations evaluated at each sweep of the Gibbs sampler approximates 
the required regression estimate. Uncertainty about the regression can be approximated similarly via the evaluation of 
the variance at each iteration. 



Ciassification function estimation 

45 

[0036] In the classification model the response y takes values 0 and 1 and obviously cannot be modelled as mixture 
of nornnal distributions as before. The response should be modelled as a Bernoulli distribution with parameter e which 
is defined as 

50 Br(yl e)=:e^(1 -e)^"^ 

where 0 represents the probability of response. Note that 9 is not observed and is a latent variable. The idea is to model 
the parameter 0 with the input space as a multivariate mixture of normal distributions. However, 0 takes values in the 
range [0.1] and cannot be sensibly modelled by the normal distribution. The following transfonnation maps 9 onto the 
55 range (-<», «>) 
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5 This is the logit transformation. We can model yf wrth the input space x as a DIrichlet mixture of nomials in the following 
hierarchical model: 



10 



15 



Ki - G 

G - D{0Go) 

[0037] As before the conditional distribution of logit 9^^^ given input variables, x^^^, conditional on n* and the allo- 
cations Is given by 



20 



where f7o,<7y,/'o ^^y ^^^^ ^^^^ regression example described above. 

25 [0038] Thus, this model fits a linear regression model on \if„^^ within each cluster which is equivalent to a logistic 
regression on y^+i in each cluster. 

[0039] The corresponding classification probability is given by 



3a 



35 



= 1 ^ nJ<^o(^nM'V n^^^^n^^)+L ^(Vn+i^^n+v "))Wn+i The /c+1 integrals 



in the above equation cannot be evaluated analytically, however Simpson's approximation works well in one dimension 
40 and can be used as an approximation method. 

[0040] The values of yf are uni(nown and must be integrated out by sampling for theni in our Gibbs sequence. Given 
an allocation of customer / to cluster / we have 



45 



(X;,\|/,.)~ A/fc:,l(X,.,\|f,.|Hy,Xy) 

[0041] If we partition (x/, y/i) into X/ and the corresponding partitions of and X / are 



50 



^7 = 



^1 ^^2 



then the conditional density of yff given x/ is univariate nomial with mean and precision given respectively by 
[0042] The full conditional density of v^/ which we are required to sample from is given by 
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p(v,- 1 //. n). ^y-. 5/ = y) « piVi iv/)p(v/ 1 u), = y) 



5 




[0043] it is straightforward to sampie from this density via rejection sampling. 
Extending to discrete input variables 

15 

[0044] The idea of introducing latent variables to model discrete target variables can be simply extended to model 
discrete explanatory variables. A general classification model is outlined with n continuous explanatory variables x, one 
discrete explanatory variable which we denote by x" and a binary response y. Thus the model, is given by the following 
hierarchical structure 

20 

(y/IV,)- Br(y,.ie,.).(x,H,)~ ^(x/IC/) 
(X/. V,., 4,. I n,-, Xf) - (^/ V/. 4. • H/. ^y) 

25 It} - G 

G - DWGo) 

where Ms a discrete probability distribution and Is a transformation of Q such that 5/ takes values in the range (-oo, 
30 oo). In the case of the binary response, described previously, f was the Bernoulli distribution and the parameter of the 
Bernoulli distribution was transformed by the logit transformation. We can use exactly the same idea for a binary input 
variable. If x' is count data, a sensible choice for A would be the Poisson distribution. The parameter of the Poisson dis- 
tribution, Q , takes values in the range (0, oo). if we take the transfonnation 4 = log C . then ^ will take values in the range 
(-00, oo) as required. 

35 [0045] To make a prediction for a new customer with attributes given by x , we use the following method. At each 
iteration of the sampler, x will fall into one of the groups and associated with that is a value of 9. For the /-th group this 
value is 

a+/7,| 

We define e^^\..., O^'^ to be the samples of 0 for that value of x. Then, the prediction for the propensity for x is . 

45 ' N 



50 [0046] The multivariate nomial distribution of dimension k is defined as 

55 

[0047] The multivariate Student-t distribution of dirjiension k is defined as 



9 



3/23/2007, EAST Version: 2.1.0.14 



EP1 085 429A1 



r(Y/2)(Y7()'"^ ^ 
s [0048] The Wishart distribution of dimension k is defined as 

lV/,(xi a. P) =n*"'-"'* n ^^)^' exp {-fr(Px)) " 

[0049] Referring now to. Figure 3, there is shown a series of data processing steps starting from a start step 30 and 
passing to the next step 31 in which the data in the data base of Figure 1 is initialised by means of a clustering algorithm 
to fonri clusters of customers. Any of a number of suitable known algorithms may be used for the purpose of step 31 . 

15 The clusters fornied in the step 31 are normal distributions. The following step 32 includes estimating the parameters 
of the normal distributions by the mean and variance/covariance of the customer attributes assigned to each cluster. 
[0050] Step 33 is the first of a series of steps in an iterative sequence. \n step 33, the result of each iteration is used 
to reassign customers to an existing normal distribution or to a new nomnal distribution. Step 34 is a sampling step to 
sample the parameters of the normal distributions conditioned on the customers assigned to them. Continuing in the 

20 iterative sequence, step 35 is a sampling step for latent values and step 36 is a sampling step for missing values. In step 
37 a calculation is made of the expected propensity of customers of interest and in step 38 a test is made to determine 
whether the iterative sequence has reached convergence. If no the iteration is continued back to step 33 and if yes the 
iterative sequence is stopped. 

[0051] Reverting now to step 35, the output from this step leads to a step 39 in which a logit transformation is per- 
25 formed on a latent parameter 9 representing the probability of response, so as to model the latent parameter as a mul- 
tivariate mixture of normal distributions. The transformation is performed as already explained in relation to the 
mathematical basis for the invention. 

[0052] As already explained, since we are modelling the input space, missing values are straightforward to impute 
and sampling for the missing values is performed in step 36. 
30 [0053] The invention can be extended to model the time to an event (referred to in statistics as survival analysis) 
using the exponential distribution. The parameter of the exponential distribution takes values on the real line. The log 
transformation of this parameter could be modelled jointly with the input data as a Dirichlet mixture of multivariate nor- 
mal distributions in the framework described. 

[0054] What has been described is a method and apparatus offering extremely flexible ways of modelling data and 
35 capable of dealing with a highly varied customer base that does not follow a global trend. It is unaffected by outliers, 
such as errors in the data or small groups of customers displaying unusual behaviour. It can also deal elegantly with 
missing values in the data. An important technical contribution of the invention is that it allows inferences to be reversed 
from a given behaviour to demographic descriptions of customers that perform that behaviour. In other words it can pro- 
vide a previously unobtainable executive summary describing the factors that drive customer behaviour. 

40 

Claims 

1. A method of classifying discrete data in a database comprising the steps of; 

45 jointly estimating the density in the covariate and response variables constituting multivariate normal distribu- 

tions, 

sampling responses from the estimate, and 

performing an artaitrary transformation on latent parameters of a k-dimensional distribution, where each dimen- 
sion Is defined on a real line (I.e - <» to -h <») and where k is an integer greater than or equal to 2. 

50 

2. A method as claimed in claim 1 . in which the step of Jointly estimating the density in the covariate and response 
variables constituting multivariate nomnal distributions uses the Dirichlet process mixtures of nonnals model. 

3. A method as claimed in claim 1 or 2, in which the step of perfonning an arbitrary transformation comprises perform- 
55 ing a logit transformation on a latent parameter representing the probability of response, so as to model the latent 

parameter as a multivariate mixture of normal distributions 

4. A method as claimed In claim 3, wherein the logit transformation is perfomied on a parameter of a Bernoulli distri- 
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bution which models the response. 

5. A method as claimed in claim 1 or 2, in which the step of perfomriing an arbitrary transformation comprises perform- 
ing a log transformation on a parameter of a Poisson distribution which models the response. 

6. A method as claimed in any one of the preceding claims in which the data is representative of the attributes of cus- 
tomers of a business organisation. 

7. Data processing apparatus for classifying discrete data in a database the apparatus comprising; 

estimating means for jointly estimating the density in the covariate and response variables constituting multi- 
variate normal distributions, 

sampling means for sampling responses from the estimate, and 

transformation means for performing an arbitrary transfonmation on latent parameters of a k-dimensional dis- 
tribution, where each dimension is defined on a real line (i.e - oo to + ») and where k is an integer greater than 
or equal to 2. 

8. Data processing apparatus as claimed in claim 7, wherein the estimating means uses the Dirichiet process mix- 
tures of normals model. . 

9. Data processing apparatus as claimed in claim 7 or 8 wherein the transformation means performs a logit transfor- 
mation on a latent parameter representing the probability of response, so as to model the latent parameter as a 
multivariate mixture of normal distributions. 

10. Data processing apparatus as claimed in claim 9, wherein the logit transformation is perfomied on a parameter of 
a Bernoulli distribution which models the response. 

11. Data processing apparatus as claimed in claim 7 or 8, wherein the transformation means performs a log transfor- 
mation on a parameter of a Poisson distribution which models the response. 

12. Data processing apparatus as claimed in any of claims 7 to 11, comprising a store of the data in the fornn of 
attributes of customers of a business organisation. 

13. A computer program adapted for programming data processing apparatus to classify discrete data in a database 
the program comprising; 

instructions for jointly estimating the density in the covariate and response variables constituting multivariate 
nonnal distributions, 

instructions for sampling responses from the estimate, and 

Instructions for perfonning an arbitrary transformation on latent parameters of a k-dimensional distribution, 
where each dimension is defined on a real line (i.e - « to + w) and where k is an Integer greater than or equal 
to 2. 

14. A method of classifying discrete data in a database, the method being substantially as hereinbefore described with 
reference to the accompanying drawings. 

15. Data processing apparatus for classifying discrete data in a database, the apparatus being substantially as herein- 
before described with reference to the accompanying drawings. 
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